Search for: All records

When text datasets are very large, manually coding line by line becomes impractical. As a result, researchers sometimes try to use machine learning algorithms to automatically code text data. One of the most popular algorithms is topic modeling. For a given text dataset, a topic model provides probability distributions of words for a set of “topics” in the data, which researchers then use to interpret meaning of the topics. A topic model also gives each document in the dataset a score for each topic, which can be used as a non-binary coding for what proportion of a topic is in the document. Unfortunately, it is often difficult to interpret what the topics mean in a defensible way, or to validate document topic proportion scores as meaningful codes. In this study, we examine how keywords from codes developed by human experts were distributed in topics generated from topic modeling. The results show that (1) top keywords of a single topic often contain words from multiple human-generated codes; and conversely, (2) words from human-generated codes appear as high-probability keywords in multiple topic. These results explain why directly using topics from topic models as codes is problematic. However, they also imply that topic modeling makes it possible for researchers to discover codes from short word lists. 

Coding data—defining concepts and identifying where they occur in data—is a critical aspect of qualitative data analysis, and especially so in quantitative ethnography. Coding is a central process for creating meaning from data, and while much has been written about coding methods and theory, relatively little has been written about what constitutes best practices for fair and valid coding, what justifies those practices, and how to implement them. In this paper, our goal is not to address these issues comprehensively, but to provide guidelines for good coding practice and to highlight some of the issues and key questions that quantitative ethnographers and other researchers should consider when coding data. 

A key goal of quantitative ethnographic (QE) models, and statistical models more generally, is to produce the most parsimonious model that adequately explains or predicts the phenomenon of interest. In epistemic network analysis (ENA), for example, this entails constructing network models with the fewest number of codes whose interaction structure provides sufficient explanatory power in a given context. Unlike most statistical models, however, modification of ENA models can affect not only the statistical properties but also the interpretive alignment between quantitative features and qualitative meaning that is a central goal in QE analyses. In this study, we propose a novel method, Parsimonious Removal with Interpretive Alignment, for systematically identifying more parsimonious ENA models that are likely to maintain interpretive alignment with an existing model. To test the efficacy of the method, we implemented it on a well-studied dataset for which there is a published, validated ENA model, and we show that the method successfully identifies reduced models likely to maintain explanatory power and interpretive alignment. 

While there has been much growth in the use of microblogging platforms (e.g., Twitter) to share information on a range of topics, researchers struggle to analyze the large volumes of data produced on such platforms. Established methods such as Sentiment Analysis (SA) have been criticized over their inaccuracy and limited analytical depth. In this exploratory methodological paper, we propose a combination of SA with Epistemic Network Analysis (ENA) as an alternative approach for providing richer qualitative and quantitative insights into Twitter discourse. We illustrate the application and potential use of these approaches by visualizing the differences between tweets directed or discussing Democrats and Republicans after the COVID-19 Stimulus Package announcement in the US. SA was integrated into ENA models in two ways: as a part of the blocking variable and as a set of codes. Our results suggest that incorporating SA into ENA allowed for a better understanding of how groups viewed the components of the stimulus issue by splitting them by sentiment and enabled a meaningful inclusion of data with singular subject focus into the ENA models. 

Quantitative ethnographic models are typically constructed using qualitative data that has been segmented and coded. While there exist methodological studies that have investigated the effects of changes in coding on model features, the effects of segmentation have received less attention. Our aim was to examine, using a dataset comprised of narratives from semi-structured interviews, the effects of different segmentation decisions on population- and individual-level model features via epistemic network analysis. We found that while segmentation choices may not affect model features overall, the effects on some individual networks can be substantial. This study demonstrates a novel method for exploring and quantifying the impact of segmentation choices on model features. 

Quantitative ethnographers across a range of domains study complex collaborative thinking (CCT): the processes by which members of a group or team develop shared understanding by making cognitive connections from the statements and actions of the group. CCT is difficult to model because the actions of group members are interdependent—the activity of any individual is influenced by the actions of other members of the group. Moreover, the actions of group members engaged in some collaborative tasks may need to follow a particular order. However, current techniques can account for either interdependence or order, but not both. In this paper, we present directed epistemic network analysis (dENA), an extension of epistemic network analysis (ENA), as a method that can simultaneously account for the interdependent and ordered aspects of CCT. To illustrate the method, we compare a qualitative analysis of two U.S. Navy commanders working in a simulation to ENA and dENA analyses of their performance. We find that by accounting for interdependence but not order, ENA was not able to model differences between the commanders seen in the qualitative analysis, but by accounting for both interdependence and order, dENA was able to do so. 

While quantitative ethnographers have used epistemic network analysis (ENA) to model trajectories that show change in network structure over time, visualizing trajectory models in a way that facilitates accurate interpretation has been a significant challenge. As a result, ENA has predominantly been used to construct aggregate models, which can obscure key differences in how network structures change over time. This study reports on the development and testing of a new approach to visualizing ENA trajectories. It documents the challenges associated with visualizing ENA trajectory models, the features constructed to address those challenges, and the design decisions that aid in the interpretation of trajectory models. To test this approach, we compare ENA trajectory models with aggregate models using a dataset with previously published results and known temporal features. This comparison focuses on interpretability and consistency with prior qualitative analysis, and we show that ENA trajectories are able to represent information unavailable in aggregate models and facilitate interpretations consistent with qualitative findings. This suggests that this approach to ENA trajectories is an effective tool for representing change in network structure over time. 

Rapid advances in technology also come with increased training needs for people who engineer and interact with these technologies. One such technology is collaborative robots, cobots, which are designed to be safer and easier to use than their traditional robotic counterparts. However, there have been few studies of how people use cobots and even fewer identifying what a user must know to properly set up and effectively use cobots for their manufacturing processes. In this study, we interviewed nine experts in robots and automation in manufacturing settings. We employ a quantitative ethnographic approach to gain qualitative insights into the cultural practices of robotics experts and corroborate these stories with quantitative warrants. Both quantitative and qualitative analyses revealed that experts put safety first when designing and monitoring cobot applications. This study improves our understanding of expert problem-solving in collaborative robotics, defines an expert model that can serve as a basis for the development of an authentic learning technology, and illustrates a useful method for modeling expertise in vocational settings. 

For many outside the profession, teaching looks simple and straightforward; however, for those working in classrooms, it can be a challenging task. In this paper we argue that teaching is a complex profession that requires both novice and expert educators alike to engage students in sets of activities aimed at transforming their understanding of a subject area. This work requires complex planning, enacting instruction, and reflecting on outcomes. In a moment to moment basis teachers must make decisions and iterate on previously made decisions in order to provide effective opportunities for students to engage with the materials, skills or content to be learned. In this paper, we aim to highlight the complexity of the decision-making process and, in doing so we make the argument that individual teachers’ decisionmaking draws upon a personal epistemic frame which includes factors such as skills, knowledge, identity, values, and epistemology. We provide examples of previous research efforts that have attempted to explore such factors and the limitations, both philosophical and methodological shortcomings of such attempts. Finally, we propose that the use of Quantitative Ethnography and Epistemic Frame Theory provides new opportunities to interrogate teachers’ practices and decision-making as a way to better understand the complexity of teacher work.